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Abstract 

We prove a lower bound expressed in the increment sequence on the 
average-case complexity (number of inversions which is proportional to 
the running time) of Shellsort. This lower bound is sharp in every case 
where it could be checked. We obtain new results e.g. determining 
the average-case complexity precisely in the Yao-Janson-Knuth 3-pass 
case. 


1 Introduction 

The question of a tight general lower bound or upper bound on the average- 
case complexity of Shellsort (due to D.L. Shell [12]) has been open for more 
than five decades [5]. We use “average” throughout in the sense of “arith¬ 
metic mean,” and the average-case complexity is the average-case of the 
number of inversions. (The number of inversions is proportions to the run¬ 
ning time, and the number of comparisons in p-pass Shellsort of n keys is 
np larger than the number of inversions.) We present an average-case lower 
bound on the number of inversions for a p-pass Shellsort with increments 
hi,h 2 , ■ ■ ■ ,hp for every number of passes and increment sequences. 

Shellsort sorts in situ a list of n elements in p passes using a sequence of 
increments hi,... ,hp with n > hi > ■ ■ ■ > hp. In the fcth pass the main list is 
divided in hk separate sublists of length n/hk, where the jth sublist consists 
of the elements in positions j mod hk of the main list (j = 1,... ,hk). Every 
sublist is sorted using a straightforward insertion sort. The efficiency of the 
method is governed by the number of passes p and the selected increment 
sequence hi,... ,hp satisfying Zip = 1 to ensure sortedness of the final list. 
Shellsort is used for example in the C standard library, in the uClibc library, 
Linux kernel, and bzip2 compressor m- 
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1.1 Previous Work 


Let log denote the binary logarithm. The original logn-pass increment se¬ 
quence [n/2j, [n/4j,...,l of Shell [12] uses worst case 0(n^) number of 
inversions, but Papernov and Stasevich [2| showed that another related in¬ 
crement sequence uses worst-case number of inversions 0(n^/^) and Pratt 
m extended this to a class of all nearly geometric increment sequences 
and proved this bound was tight in the sense of coinciding with the upper 
bound. Incerpi and Sedgewick [2| constructed a family of 0(logn)-length 
increment sequences for which Shellsort runs in number of 

inversions, for all e > 0. Plaxton, Poonen and Suel |9| proved an 
lower bound for p passes of Shellsort using any increment sequence, for every 
e > 0 and showed that this bound is tight for p = P(logn). Since every pass 
takes at least n steps this shows an P(nlog^ n/(loglogn)^) lower bound on 
the worst-case of every Shellsort increment sequence. The currently best 
asymptotic method was found by Pratt m- It uses all log^ n increments 
of the form 2*3-^ < [n/2j to obtain number of inversions O(nlog^n) in the 
worst case. Moreover, since every pass takes at least n steps, the average- 
case complexity using Pratt’s increment sequence is 0(nlog^n). Knuth [5] 
shows 0(n^/^) for the average-case case of p = 2 passes and Yao |15] derives 
an expression for the average case for p = 3 that gives not (yet) a direct 
bound but was used by Janson and Knuth to derive an upper bound of 
q(j^ 23/15) average-case complexity of 3-pass Shellsort. In |7] Jiang , 

Li and Vitanyi derived a general lower bound of on the average- 

case complexity of p-pass Shellsort. This lower bound shows that the only 
possibility of Shellsort to run on the average in 0(nlogn) inversions is for 
the number of passes p to satisfy p = 0(logn). Apart from this, no nontriv¬ 
ial results were known for the average case until the results presented here. 
A more detailed history can be found in |5|. 

1.2 Present Work 

We show a lower bound on the average number of inversions (which is the 
same order of magnitude as the running time and the number of compar¬ 
isons) of Shellsort expressed in the increment sequence used (Theorem [1]) . 
The proof uses the fact that most permutations of n keys have high Kol¬ 
mogorov complexity. Since the number of inversions in the Shellsort process 
is not easily amenable to analysis, we analyze a simpler process. This sim¬ 
pler process has at most as many inversions (the sum of the minor sequence 
in Definition dj) as the original process. Hence a lower bound on the number 
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of inversions in the simpler process is a lower bound on the number of inver¬ 
sions of the original process. We show that the largest number of inversions 
of each key in the /cth pass of the simpler process is less than hk-i/hk where 
hi,... ,hp is the increment sequence and ho = n (Claim [2|). Subsequently 
it is shown using the high Kolmogorov complexity of the permutation that 
most keys in each pass have a number of inversions close to the maximum. 
This gives a lower bound on the total number of inversions of the simpler 
process (Claim [3|) and hence a lower bound for the original process. This 
holds for the chosen single permutation. Since all permutations but for a 
vanishing fraction (with growing n) have this high Kolmogorov complexity, 
the lower bound on the total number of inversions holds for the average 
case of the original Shellsort process (Theorem [1]). The lower bound seems 
to be tight since it coincides with the known bounds. For 2-pass Shellsort 
Knuth in [5] determined the average-case complexity and the new lower 
bound on the average complexity coincides (Corollary [T]). For 3-pass Shell- 
sort Knuth and Janson [3], building on the work of Yao m , gave an upper 
bound on the average-case complexity and the new lower bound coincides 
(Corollary [T]). This yields the new result that the average-case complexity 
of Shellsort for this increment sequence is now determined. They [3] conjec¬ 
ture an upper bound on the average-case complexity for another increment 
sequence. The lower bound on the average-case complexity established here 
for this sequence coincides with this upper bound (Corollary [T]). For the 
logarithmic increment sequences of Shell [12], Papernov and Stasevich [8], 
Hibbard [l], and Pratt m also reported in [5], the lower bound on the 
average-case complexity for the respective increment sequences is H(nlogn) 
(Corollary [2|). No upper bound on the average-case complexity is known 
for any of these increment sequences. For the square logarithmic increment 
sequence of Pratt |10] the average-case complexity is known. Again, the 
lower bound given here coincides (Corollary [3|). 

2 The Lower Bound 

A Shellsort computation consists essentially of a sequence of inversion (swap¬ 
ping) operations. We count the total number of data movements (here in¬ 
versions). Keys in the input permutation go by inversions to their final 
destination. The sequences of inversions constitute insertion paths. The 
proof is based on the following intuition. There are n! different permuta¬ 
tions of n keys. Given the sorting process (the insertion paths in the right 
order) one can recover the original permutation from the sorted list. The 
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sum of the lengths of the insertion paths must be at least as large as the 
length of a most concise representation of the starting permutation. 

Theorem 1. The average-case number of inversions in a p-pass Shellsort 
algorithm on n keys with increment sequence hi,... ,hp, and denoting hn = 
n, has a lower bound of Tl {nYf/k=i hk-i/hk)- 

Proof. Let the list to be sorted consist of a permutation tt of the keys 
1, ..., n. Let T be a p-pass Shellsort algorithm with increments {hi,..., hp) 
such that hk is the increment in the kth pass and hp = 1. Denote the 
original permutation by tt = tto and the permutation resulting from pass k 
by TTfc. In each permutation the keys are ordered left-to-right. In the final 
permutation Tip = 12... n the least key 1 is on the left end and the greatest 
key n is on the right end. 

For k = 1,2,... ,p, the kth pass starts from iTk-i and this list (or permu¬ 
tation) is divided into hk separate sub lists or hk-chains of length, n/hk, where 
the hth /ife-chain {1 <h < hk) consists of the keys in positions j mod hk = h 
of the main list 7Tk-i {j = 1,... ,n). The insertion sort of a /i^-chain goes 
as follows. We start at the left end. If the second key is less than the first 
key then the second key is swapped with the first key. Otherwise nothing 
happens. This creates a new /ifc-chain. If the third key is smaller than the 
first key or the second key in the new /i^-chain, then the third key is inserted 
in its correct position in the <-order before the first key or in between the 
first key and the second key. Otherwise nothing happens. We continue this 
way. The ith key is inserted in its correct position in the <-order in the ini¬ 
tial segment of the current /i^-chain consisting of the first key through the 
{i — l)th key. All keys greater than the ith key in this initial segment move 
one position to the right. This is possible since the inserted key left a gap at 
the ith position of the current /i^-chain. An inversion is a swap of key i with 
key j which changes list .. .ij ... to list ... ji.... We can view the insertion 
above as the ith key changing place with the key before it (an inversion), 
then changing place with the key before that (a second inversion), and so 
on, untill it ends up in its correct position. The inversions it had to make 
to do so is called its insertion path. By the time the final key is inserted in 
its correct position in the <-order the /i^-chain involved is sorted. 

All keys i = 1,2,... ,n reside in an /ifc-chain. Let mi^k be the number of 
inversions of key i in its /ifc-chain in this sorting process. At the end of the 
sorting process the /i^-many /i^-chains are merged to establish permutation 
TTfc by putting the jth key of the hth sorted /i^-chain into position h+{j—l)hk 
of permutation (1 < /i < hk). That is, the jth position of a /i^-chain has 
the same position in permutation TTk-i as it has in permutation TTfc. The 


4 



sum 


( 1 ) 


n p 

i=l k=l 

is the total number of inversions that algorithm A performs. 

Definition 1. Let n, vr, the increment sequence hi,...,hp = 1 and the 
Shellsort algorithm be as described above. A simple Shellsort algorithm is 
a Shellsort algorithm where the insertion sort of the /i^-chains is changed. 
For all keys i = 1, 2,... , re and passes k = 1, 2,... ,p, when it is the turn of 
key i in position j of its /ifc-chain, it does the following. It moves a number 
k,k, 0 < li^k < n/hk — j, to the right by inversions subject to the following 
condition. The sequence li^i ,..., ln,p is such that pass 1 starts with vr and 
pass p finishes with the sorted list of keys 12... re. The minor sequence 
Sn = ..., nn,p is the lexicographic in i, A: (1 < i < re and 1 <k <p) the 

first sequence of a simple Shellsort algorithm such that T' = ^11=1 
is least over all X^ILi sequences ■ ■ ■ ,ln,p associated with 

simple Shellsort algorithms. Denote Tj = Y%=i all 1 < i < re such 

that T'= T,. 

Claim 1. Given n,A and all ’s in appropriate fixed order, we can com- 
putably reconstruct the original permutation vr. 

Proof. Let the sequence of permutations resulting from the minor sequence 
he TT = pq, pi,..., pp = 12 ... re. The re^^p’s in the appropriate order specify 
the initial permutation pp^i of pass p. For k = p,p — 1,... ,1 given pk we 
can in this way reconstruct the initial permutation of pass k. Hence given 
Pp = 12... re we can reconstruct the original permutation po = tt given the 
data items in the claim. □ 

Claim 2. (i) T' < T. 

(ii) < hk-i/hk for all 1 < f < re, 1 < A: < p. 

(iii) For every z (1 < f < re) holds that from Ti = 'Y%,=i '^i,k one can 
extract the rej^^’s in the order nip ,..., Ui^p. 

Proof, (i) By Definition [1] each Shellsort sequence mip ,..., mn,p is a simple 
Shellsort sequence. Hence by Definition [1] and the minimality of the minor 
sequence item (i) holds. 

(ii) Let the sequence of permutations of the keys resulting from pass 
1 through p using the Ui^s (1 < i < re, 1 < k < p) he pi,..., pp = 
12 ... re, respectively. Assume by way of contradiction that there exist i, k 
{I < i < n, 1 < k < p) with i + k least such that nip > hk-ijhk. (In 


5 


the following the rounding is ignored.) Define '■= + 1 and 

•= - hk-i/hk, while = rij^h otherwise. If > hk- 2 /hk-i 

then := + 1 and - hk- 2 /hk-i, while 

otherwise. This process is repeated as often as necessary. However, it is not 
possible that it goes all the way to the 1st pass and results in > h^jhx. 

Namely this would imply that key i moves to a position greater than n//ii 
in its hi-chain of length njhx which is impossible. Assume that the process 
above is repeated I < k times. The resulting permutations of the passes 
are pi,..., pk-i-idk-i ■ ■ ■ crkPk-i ■ ■ ■ Pp where ai_k-, ■■■,crk are possibly new 
permutations of the n keys. However for / > 0 the Y17=i Y%=i ^ 
contradicting the minimality of the minor sequence ni^i,..., nn,p- 

(iii) Let 1 < i < n. The number of inversions which move key i from its 
position as the jth key in tt from the left end to its final position as the zth 
key from the left end in = 12 ... n is Tj = Y%=i 'k^i,k with max{j — 0} = 

Z]fc=i That is, max{j — i,0} is represented in a mixed radix system 

with radices hi,h 2 , ■ ■ ■ ,hp. The question asked is whether this representation 
is unique or not. By the minimality of T' all Ti,..., Tp are minimal. Hence in 
each n^p,..., rii^p the Ui^kS are as great as possible subject to rii^k < hk-i/hk 
(item (ii)) and Tj = Yl^k=i 'k^i^k^k in the order A: = 1,... ,p, for each 1 < i < n. 
Since Hq > hi > ■ ■ ■ > hp this shows the representation is unique. □ 

There are n! permutations vr of n keys to be sorted. Below we use the 
plain Kolmogorov complexity defined by Kolmogorov in [1] and denoted 
by C in the text [6]. Let x,y,z be natural numbers. Here is used that 
C{x\y) < logx + 0(1) for all x and y and C{z\y) > logx — c for fixed y 
and \bin{x)\ — |6in(x)|/2‘^ numbers z ^ x such that \bin{z)\ = \bin{x)\. Here 
bin{x) is the binary representation of x and \bin{x)\ is its length in bits. 
Iterating the use of standard pairing functions, such natural numbers may 
consist of fixed finite sequences of natural numbers. 

There are re! ~ \/27rre(g)"’ permutations of re keys (here vr is for once 
the number vr). By Stirling’s approximation used below we are justified to 
choose the permutation vr with Kolmogorov complexity 

(^(vrlre, A, P) > relogre — 3re, (2) 

with A the algorithm used in this p-pass Shellsort (including the increment 
sequence), and P a constant-size algorithm to process all the information 
and to output vr. 

Denote the minor sequence rei^i,...,ren,p hy Sn = Ti,T 2 ,... ,Tp. A com¬ 
putable description of Sn, given re, A and Q (an 0(1) bit program included 
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in P above), requires at most 


n p 

\descr {Sn)\ = Ui^k) + D (3) 

1=1 k=l 

bits where D is the number of bits required to be able to parse the main 
part of descr{Sn) into its constituent parts, that is, the concatenated bit 
strings rii^k of length log l<fc< p). By Claim [1] we can 

compute permutation vr from descr{Sn), given n,A and Q. Hence 

|descr(S'„)| > C{TT\n, A, P). (4) 

From ([2]) and ([1|) it follows that 

|(iescr(S'n)| > nlog(n/8). (5) 


Claim 3. Writing ho = n we have 


ni,k = H n hk-i/kk 


i=l k=l 


k=l 


Proof. In pass 1 for every key i (1 < i < n) we have < nfhi. In general 
by Claim [2] item (ii) for every pass k {1 < k < p) rii^k < hk-i/hk- Since 
0^=1 kk-i/hk = ho/hp = n we have by ([3]) and ([5]) that 


\descr{Sn)\ 

n 


Z]fc=i^(log(^fc-i/^fc) - Qfc) p_ 

n n 

P 7-^ 

1 ^ 1 ^ 

logn - > Ofc H-> log-, 

n 8 

k=l 


( 6 ) 


where Ok = log{hk-i/hk) - 1/n XlILi log for A; = 1,2,... and > 0 
by Claim [2] item (ii). 

We show that D/n = o(logn). To be able to parse descr{Sn) into 
its constituent descriptions descr{Ti),... ,descr{Tn) it suffices that D = 
21og |fiescr(ri)| +0(1). Since Tj = Ylk=i'^i,k < n we have D < 
2n log log n+0(l) (1 < i < n). Namely, to encode every part Tj for 1 < i < n 
such that it can be parsed from the total we double each bit of the part except 
the last bit which is followed by its complement. This doubles the length of 
each part. Additionally we require 0(1) bits for a program to retrieve the 
Tj’s and extract ..., rii^p from them. This can be done in a unique way 
by Claim [2] item (iii). The total of the description D is o(n log n) bits. 
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Hence up to lower order terms the last inequality of ([6|) is rewritten as 
— 3- Since > 0 for every 1 < k < p we have < 3. Writing 
out and reordering this gives up to lower order terms 

n 

\og{hk-i/hk) < l/n^loguj^fc+ 3, 

i=l 

and by exponentiation of both sides of the inequality one obtains 

n 

hk-i/hk = 

2=1 


By the inequality of the arithmetic and geometric means and rearrang¬ 
ing we obtain XlILi = ^{nhk-i/hk) for every 1 < A: < p. Therefore, 

ELi EEi ni,k = ^{n ELi hk-ilhk). □ 


Since T' < T hy Claim [2] item (i), a lower bound for T' is also a lower 
bound for T. Therefore Claim [3] proves the statement of the theorem for the 
particular permutation vr. 

By Stirling’s approximation logn! ~ nlog(n/e)-|-^ logn-l-O(l) nlogn— 
1.44n -|- ^logn -|- 0(1). Therefore nlogn — 1.5n < logn! < nlogn — n for 
large n. Hence by [6l Theorem 2.2.1] which uses a simple counting argu¬ 
ment, at least a (1 — l/n)-fraction of all permutations vr on n keys satisfy 
m- Since 1/n —>• 0 for n ^ oo, and for all permutations vr on n keys we 
have T = 0{n^), a (1 — l/n)th fraction of all n! permutations has a lower 
bound as permutation vr does, and a 1/nth fraction has a lower bound of 
at least 14(0) and at most 14(n^). Hence the lower bound on the average 
number of inversions. □ 


Corollary 1. For p = 2 with hi = n^/^, and = 1 this yields 


T = f4(n(n^"^/^ + n^/^) = 14(n^/^), 


which coincides with the best number of inversions for 2-pass Shellsort T = 
0(n5/3) Tjging the same increment sequence hi = n^/^,/i 2 = 1 as given by 

0 . 

For p = 3 with hi = h 2 = n^/®, and hs = 1 this yields 

T = 14(n(ni-"/i5 + n^/i^-Vs + ^i/5) ^ ^(n'+s/iS) ^ 

The upper bound of for 3-pass Shellsort using the same increment 

sequence hi = 0(n^/^®), /i 2 = 0(n^/®), h^ = 1 with the additional restriction 
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that gcd{hi, /i 2 ) = 1 is given in [3]. This reference uses a complicated prob¬ 
abilistic analysis based on the still more complicated combinatorial charac¬ 
terization in m- Together with the lower bound we establish the new fact 
that the average number of inversions of 3-pass Shellsort with this increment 
sequence is 

In Section 10 of [3] it is conjectured that with hi « and /12 ~ 

(/i 3 = 1 ) one may obtain an average-case number of inversions of 
Using the theorem above shows that T = + 77 , 1 / 2 - 1/4 _|_ 77 , 1 / 4 ^ _ 

Therefore, if the conjecture on the upper bound is true then 3- 
pass Shellsort has an average-case number of inversions of for this 

increment sequence. 

Corollary 2. The increment sequence hi,... ,hp with p = [lognj of Pa- 
pernov and Stasevich in [5] is hi = n/2 + l,h 2 = n/2^ + 1,... ,hp = 
-|-1. The worst-case number of inversions reported by [ 8 ] is 0(n^/^). 
Since hk-i/h^ ~ 2 and 2 = U(l), the theorem above gives a lower bound on 
the average number of inversions of T = ^(1)) = fl(nlogn). 

The increment sequence of Hibbard [l] with increment sequence 2^ — 
1 until it passes n has a worst-case number of inversions 0(n^/^). With 
a similar analysis as before it has a lower bound on the average-case of 
T = H(nlogn). It is conjectured to lead to an average-case number of 
inversions of 0 (n®/^) in m reported in [5]. This conjecture is difficult to 
settle empirically. For n = 100,000 we have logn ~ Hence we need 

to do many experiments for n much larger than 100,000 to obtain evidence. 
The upper bound may well be O(nlogn). 

Pratt’s logarithmic increment sequence (one of his “hypergeometric” se¬ 
quences) in [To] also reported by [5] is hi,..., hp with hk = (3^ — l )/2 not 
greater than \ri\. This increment sequence leads to a worst-case number 
of inversions of 0(n^/^). In this case hk-i/hk ~ 3 and 3 = H(l) and the 
number of passes is p = log 3 n. The theorem above gives a lower bound on 
the average number of inversions of T = ^(1)) = H(nlogn). 

The original increment sequence used by Shell m was [n/ 2 j, [n/ 2 ^J, 
and so on for logn passes. Knuth [5] remarks that this is undesirable when 
the binary representation of n contains a long string of zeroes. It has the 
result that Shellsort runs in worst-case time 0(n^). Since this increment 
sequence satisfies the same analysis as the above one of | 8 ] the lower bound 
on the average number of inversions is H(nlogn). 

By |7| the average number of inversions of Shellsort can be 0(nlogn) 
only for an increment sequence hi,... ,hp with p = 0(log n). We have shown 
here that the lower bound on the average inversions is H(nlogn) for many 
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increment sequences of this length. It is an open problem whether it can 
be proved that for some such increment sequence the average number of 
inversions is 0(nlogn). 

Corollary 3. For Pratt’s square logarithmic increment sequence hi,... ,hp 
with p = 0((log n)^), the average-case number of inversions is lower bounded 
by T = ^^(1)) = n(n(logn)^). The precise average-case 

number (and worst-case number) of inversions is 0(n(logn)^) in |10] . and 
therefore the lower bound is tight. 

3 Conclusion 

The lower bound on the average case number of inversions of Shellsort us¬ 
ing p passes in [7] is for worst-case increment sequences. Here we gave a 
lower bound on the average-case complexity for each increment sequence 
separately. This is in several cases larger than the lower bound of above 
reference. In fact, the lower bound given here seems to be tight as follows 
from the corollaries. If the increment sequence is the best possible for a 
given number of passes then the lower bound should reflect this. A tantaliz¬ 
ing prospect is to obtain from the given lower bound one which is expressed 
only in the number n of keys to be sorted and the number of passes in the 
sorting process, and which is tighter than the lower bound of [7]. 
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